Acceleration of Large Transformer Model Training by Sensitivity-Based Layer Dropping

Transformer models are widely used in AI applications such as Natural Language Processing (NLP), Computer Vision (CV), etc. However, enormous computation workload be-comes an obstacle to train large transformer efficiently. Recently, some methods focus on reducing the during training by skipping layers. How-ever, these use simple probability distribution and coarse-grained calculation, which significantly affect model accuracy. To address issue, this paper we propose a novel method accelerate training—Sensitivity-Based Layer Dropping (SBLD). SBLD uses lay-er-wise sensitivity data switch on/off layers proper order keep high Besides, adjust of with scheduler speed get faster convergence. Our results show that solves accuracy drop issue com-pared prior layer dropping methods. can decrease end-to-end time 19.67% GPT-3 Medium model, same increasing 1.65% w.r.t. baseline. Furthermore, for SwinV2-L obtained Top-1 Top-5 accuracies also higher vs. Thus, proposed is efficient practical improve training.